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ABSTRACT 

Through support of the National Science Foundation’s Department Level Reform program, En¬ 
gineering Students for the 21 st Century (ES21C) has implemented a ten-course sequence designed 
to help students develop into engineers. Spread across the Electrical and Computer Engineering 
(ECE) curriculum at Oklahoma State University, these courses were supported by engaging gradu¬ 
ate students, building infrastructure to scaffold student development, and self-reflection on what 
it means to develop as an engineer. Four case studies from the spectrum of courses illustrate the 
on-going changes. While the project is still in the process of measuring changes in student learn¬ 
ing and attitudes, preliminary project evaluation results are presented along with how formative 
evaluation has changed the project direction. From participants’ experiences, both expected and 
unexpected, some of the factors that contribute to project successes and failures are outlined. 
A key finding is that the journey of reforming undergraduate programs needs to be guided by 
knowledge of both current location and destination. A taxonomy developed under this project 
to guide, discuss, and measure reform efforts is introduced that helps faculty map pathways to 
meaningful reform. 

Keywords: department level reform, student development, engineering taxonomy 


INTRODUCTION 


"...there is nothing more difficult to take in hand, more perilous to conduct, or more 
uncertain in its success, than to take the lead in the introduction of a new order of things. 


SUMMER 2011 


1 




ADVANCES IN ENGINEERING EDUCATION 

Engineering Students for the 21 st Century: Student 
Development Through the Curriculum 


Because the innovator has for enemies ail those who have done well under the old 

conditions, and lukewarm defenders in those who may do well under the new.” 

— Nicolo Machiavelli, The Prince, chapter VI 

Decades of research on how engineering students learn has resulted in breakthroughs; there is 
consensus by experts on effective pedagogy. Although this consensus presents engineering educa¬ 
tors with unprecedented opportunities to improve student learning, effective pedagogy has entered 
the classroom at a glacial pace. Widely read reports of expert panels (National Research Council 
2004, 2005, 2007, 2009 Hyperlink to these resources ) have found that despite the progress in 
research on how engineering students learn, engineering enrollments are lagging, women are not 
significantly better represented, and under-represented groups have made little progress. In addition, 
the ballooning cost of higher education, which puts college out of reach for an increasing number 
of students, is primarily supporting administration and research rather than education (Wellman 
et al. 2009). There is increasing concern that a proportionally decreasing number of technologically 
innovative graduates will result in a future decline in economic competitiveness. 

NSF’s Department Level Reform (DLR) program explored means of fostering effective and sus¬ 
tainable change in university programs to better and more efficiently educate engineers. Unlike 
many current efforts in engineering education, the DLR program recognized that difficulties in re¬ 
forming engineering education may be structural as well as pedagogical. Department-level reform 
created experimental test-beds of alternative educational structures and methods that addressed 
pervasive problems involving a broad array of stakeholders. To succeed in this undertaking, reform 
projects attempted to change entrenched departmental cultures. As the Machiavelli quote at the 
start of this section implies, cultural change is a difficult undertaking with a long time constant; at 
meetings of DLR grantees many participants felt as if they were trying to fit ten years of work into 
a three year project. 

This paper provides a brief and transitory glimpse of changes in one degree program undertaking 
reform. Curriculum reform is multi-faceted, uncertain, and involves negotiation between participants, 
each of whom have their own beliefs and goals. Reform is also an on-going process, if this paper 
had been written a year ago some perspectives might be different, as they will be different a year 
hence. To try to capture this complex, dynamic process, this paper differs from more traditional 
engineering education papers in three ways. First, the multifaceted nature of reform made it difficult 
to choose a single theoretical lens with which to view the project. The project integrated multiple 
pedagogies and allowed faculty in individual courses to choose their own approach to reform. Thus 
the view is more like that of a bug’s eye than a camera lens; many small overlapping images need 
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to be processed to make a complete picture. To create as coherent a picture as possible from the 
efforts of many participants—some of whom did not agree with all aspects of the reform effort— 
this paper was written by two participants, with input and dissenting opinions solicited from others 
involved in the project. Second, reform, as Machiavelli understood, is a difficult endeavor with a long 
time constant. Successfully reforming a program requires participants maintain momentum against 
damping forces. The energy to keep going was supplied by the passion of participants supported 
by the positive feedback of seeing how change impacted students. This paper tries to convey some 
of this emotional energy. Third, this paper seeks to capture changes across a program rather than 
in a single course. To keep the paper to a manageable length, many courses are not discussed in 
detail. Also, the paper does not fully explore all the paths that led to dead ends. Rather to effect 
and support change in other programs, this paper focuses primarily on what has proven effective 
and constant over time. 


ENGINEERING STUDENTS FOR THE 21ST CENTURY 


This section, divided into four subsections, describes the reform project. The first section out¬ 
lines the vision at the start of the project and the basis for undertaking reform. The second section 
provides some information about the program that was reformed and the initial, planning phase 
of the reform project. Many of the initial ideas attempted in the planning phase did not work as 
planned. The third section reports on the strategy used to implement the project. The final section 
outlines the foundation in learning science by describing the theories and pedagogies the project 
is based upon. 

Beliefs, Vision, and Assumptions 

Engineering students learn what we teach them, but often do not become what we intend them 
to. The behaviors that let students succeed in classes do not always correlate with the behavior 
required to succeed in engineering careers. Engineering Students for the 21st Century (ES21C) is 
trying to align the behaviors that are taught in our program with those that help students succeed 
in engineering careers. To accomplish this goal ES21C is attempting to transition a subset of classes 
in the degree program from focusing on knowledge acquisition to emphasizing student develop¬ 
ment as engineers. 

At the heart of Engineering Students for the 21st Century is the recognition that degree program 
offered at the start of the project was primarily knowledge-based. In this knowledge-based program 
the engineering degree was defined by a specific set of content students must learn; i.e. "if you 
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learn these things you will get a degree that certifies you are an engineer." This characterization was 
certainly not monolithic throughout the department, while approximately 15% of the faculty were 
engaged in developing a broad spectrum of student skills, efforts were not coordinated or focused. 
An inherent assumption of this knowledge-based paradigm is that by learning specific content stu¬ 
dents will be able to function as engineers. Knowledge-based programs thrived in a world where the 
specialized information needed for engineering could only be found and learned at universities. In an 
age where the internet makes engineering knowledge widely available, the long term sustainability 
of this model is questionable. As Eli Noam (Noam 1995) noted more than a decade ago: “ Today’s 
production and distribution of information are undermining the university structure, making it ready 
to collapse in slow motion once alternatives to its function become possible.” 

Engineering Students for the 2ist Century seeks to make programs development-based rather 
than knowledge-based by creating a set of classes focused on developing the broad set of skills 
students need to understand problems in depth. To develop both knowledge and skills, students are 
walked through the process of solving the problem while learning concepts needed to understand 
the problem. The project posits that it is of greater importance to develop the skills needed to solve 
in-depth problems than to try to cover a large breadth of content in electrical engineering. In ES21C 
in-depth research or design projects become key experiences in a diverse curriculum. 

To engender the cultural change needed to support the shift from knowledge acquisition to skill 
development, it is critical to address not only what and how students learn but also how faculty 
support learning and the learning environment. In ES21C the role of faculty changes from lecturers 
to mentors and scholars, guiding academic development towards complex problem solving tied to 
real world problems. Development-based classes attempt to create overlap between what engineers 
need to learn (Practice of Engineering), what students are motivated by (Perception of Engineer¬ 
ing), and creating an environment that facilitates learning (Teaching Engineering) as shown in the 
Venn diagram of Figure 1. 

Under ES21C a small subset of the courses (approximately 25%) in the electrical engineering 
program were, and continue to be, modified to focus on student development. The goal of ES21C 
is to have one course each semester of the program that emphasizes student development, with 
all the development courses sharing a common strategic approach. These courses make course 
material relevant by teaching course content in the context of a project while building teamwork 
and communication skills. Within this strategic framework, each faculty member is free to choose 
topics and teaching techniques appropriate to their course and their own beliefs about learning 
and teaching. 

In summary, the vision of Engineering Students for the 21st Century is to shift the focus of an 
engineering degree from knowledge to developing the broad array of skills and knowledge needed 
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by practicing engineers. The experiences gained will hopefully contribute to the sweeping realign¬ 
ment of engineering programs in U.S. universities called for by the National Academy of Engineer¬ 
ing, National Science Foundation, and National Research Council (National Research Council 2004, 
2005, 2007, 2009 Hyperlink to these resources ). 

Context and Description of Initial Reform Plan 

The reform project took place in the School of Electrical and Computer Engineering (ECE) at Okla¬ 
homa State University (OSU). OSU is a comprehensive, land grant, research university with multiple 
campuses; the reform project took place at the main Stillwater campus. ECE is one of six engineer¬ 
ing schools in the College of Engineering, Architecture, and Technology (CEAT). During the grant 
period ECE had approximately twenty five tenure-track faculty members; many faculty were hired 
in the last decade due to a conscious effort to expand the engineering college. During the project 
period the computer engineering became separately accredited as a degree program. The student 
demographics are predominately traditional students who come both from the two urban centers 
in Oklahoma as well as many small, rural towns. Most students are Oklahoma residents. 

From a high of nearly 500 undergraduate students in 2002, enrollment in the program fell, then 
rebounded to about 300 as of early 2010; enrollment has dropped in many ECE degree programs 
nationwide. Most attrition from the program is in the freshman and sophomore years. The engi¬ 
neering degree program is divided into two phases: pre-professional school in the first two years 
and professional school in the third and fourth years. Technically, students are not members of the 
department before they enter professional school. The mean time to graduation is between nine 
and ten semesters, reflecting the fact many students work while in college. Pre-professional school 
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courses generally focus on math, science, and engineering science, are taught in large enrollment 
traditional lecture formats, and are taken by most engineering students. Professional school courses 
are more discipline specific, and taught within the department. The junior year courses form a 
“common core” taken by all students, while in the senior year student take guided electives in five 
“areas of specialization”: power and energy, computer engineering, solid state devices, signals and 
systems, and electromagnetics and photonics. 

The project begun in 2003, when the program was awarded a DLR planning grant. Since the 
engineering faculty were not then conversant in education, the project included a faculty member 
from OSU’s college of education and another faculty member from the OSU library who would man¬ 
age dissemination. During the planning phase, trial implementations of reform were made in four 
courses and participants met several times a month to brainstorm about needed changes. Much 
of the discussion in these meetings focused on ways to move the emphasis of courses away from 
“shallow” or “artificial” learning to support “deep” or “authentic” learning as discussed in (McClymer 
and Knoles 1992). The terms “shallow”, “artificial”, “deep”, and “authentic” were vivid and were clear 
to engineering faculty, who were often put off by technical education terminology. These terms 
are not generally well accepted definitions in engineering education, however. Shallow or artificial 
learning refers to students’ use of strategies like pattern matching or memorization to pass a class 
with as high a grade as possible, usually successfully. Such learning is reflected in an inability to 
transfer (Bjork 1994) learning and poor recall. Shallow learning is also restricted to a few types of 
knowledge (Krathwohl 2002), mainly factual and procedural, and is evidenced by students who have 
difficulty placing their work in a broader context. The planning phase hypothesized that making 
learning more relevant to students would improve learning outcomes since students’ perceptions 
of “artificial” were reinforced by test or homework problems that failed to address skills students 
believed engineers needed. In contrast, authentic learning sets tasks for students that mimic those 
used by practicing engineers, contextualize knowledge, and help students see how what is learned 
can be used in different contexts to support transfer. Participants generally agreed that in order 
to develop deep learning students needed to be given authentic tasks. In other words, to become 
engineers students need to continually practice being engineers. Authentic tasks may take stu¬ 
dents outside their comfort zone and demand extra effort, thus problems or projects should match 
students preconceptions of engineering work (see Figure 1). 

These almost Zen-like statements were not easy to put into effective practice. A key lesson from 
the planning phase was that students, like Aeneas and all those who seek wisdom, make potentially 
perilous journeys: “The way downward is easy from Avernus. Black Dis’s door stands open night and 
day. But to retrace your steps to heaven's air, there is the trouble, there is the toil." (Virgil 1990). 
Authentic projects have a non-negligible risk of failure, potentially lure students into blind alleys, 
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and may require tacit knowledge students don’t possess (Collins 2001). Developmentally-oriented 
classes in ES21C were thus supposed to incorporate a significant amount of student support to avoid 
some of the pitfalls often associated with simply assigning projects. A key result from the planning 
phase was the need to provide security and assurance for students. 

By the end of the two year planning phase, the vision described in the previous section had taken 
shape. The consensus of participants was that in order to successfully solve authentic problems, 
students need to both learn requisite knowledge and skills and be explicitly taught the process of 
solving the problem. Teaching the process of solving problems requires that faculty address different 
forms of knowledge and a range of cognitive processes (Krathwohl 2002) in their course. Memo¬ 
rizing Ohm’s Law, for example, represents a different type of learning than being able to measure 
V, I, or R and a different cognitive process level than analyzing how a circuit will behave as voltage 
increases. Faculty thus need to identify the types and levels of learning that match course goals, 
ensure students have or are explicitly taught necessary skills, and finally adopt pedagogies that 
support this type and level of learning. To address a wide range of knowledge Bloom’s Taxonomy 
was adopted as a guide for the reform project; the difficulties encountered and development of an 
engineering taxonomy will be discussed subsequently. 

Engineering Students for the 21 st Century’s Strategic Approach 

As discussed above, ES21C is transitioning a set of courses in the curriculum from a knowledge- 
based paradigm (acquiring a set of concepts) to being development-based (emphasizing students’ 
development) to create a more effective, engaged, and efficient program. ES21C hypothesizes that 
a development-based program will produce more effective engineers by teaching students to use a 
broad range of knowledge in the context of solving engineering problems. Since students have the 
opportunity to experience a range of skills, they can identify roles that mesh with their personal goals, 
needs, and interests and become more engaged in their studies. Efficiency—effective use of both 
institutional and human resources—arises by aligning the undergraduate degree with faculty research 
to better engage graduate students in teaching and centralizing lab facilities as discussed later. 

The original plan for program reform focused on two overarching goals which still guide ES21C: 

• Increasing the depth of student learning by restructuring ten courses in the electrical 
engineering program to focus on student development. 

• Redefining the role of faculty by engaging both current and future faculty in integrating schol¬ 
arship back into teaching. 

These two goals are obviously intertwined and synergistic; one is not possible without the other. 
The initial reform plan was to pursue both goals simultaneously. The second goal requires cultural 


SUMMER 2011 


7 






ADVANCES IN ENGINEERING EDUCATION 

Engineering Students for the 21 st Century: Student 
Development Through the Curriculum 


change, however, and has been much slower to achieve and evaluate. This paper thus focuses pri¬ 
marily on the first goal since implementation was (relatively) straightforward due to the measurable, 
concrete outcome. Progress towards the second goal is addressed briefly throughout the paper. One 
key step made to date in achieving the second goal was establishing a university-wide certificate 
program for graduate students and postdoctoral fellows focusing on faculty preparation with an 
apprenticeship component. This program has allowed graduate students to become more engaged 
in teaching and able to support faculty in the reform efforts. A significant effort was also aimed at 
faculty development by supporting workshops and travel. 

The strategy adopted by ES21C was to focus reform efforts on individual, stand-alone courses 
rather than an entire curriculum to make the reform more sustainable if faculty changed. The plan¬ 
ning phase of the project showed that a rigid, prescriptive model would not be adopted due to the 
range of faculty beliefs about effective teaching. To balance the need to accommodate a range 
of faculty beliefs with coherence of the reform program, developmental courses share a common 
structure, or strategy. A common strategy was felt to be vital (Felder, et al. 2000) since an unco¬ 
ordinated set of teaching methods forces students to continually adapt to changing expectations 
(National Research Council 2000). The strategy was guided by Bloom’s Taxonomy (Krathwohl 
2002), as discussed later. It should be noted the Bloom’s Taxonomy is not a model of student 
development, but rather a tool that permits faculty to choose appropriate learning outcomes for 
their course and students. 

The original plan for choosing the courses was based on two broad groups that faculty felt were 
critical to student development: applying mathematics to engineering problems, and system integra¬ 
tion. At the time of the award, approximately 40% of engineering students made a failing grade or 
withdrew from the introductory calculus course and faculty cited mathematics as the major shortfall 
of student preparation. A review of curricula indicated OSU students spent less time on systems 
engineering compared to peer universities. The courses reformed are shown in the table below with 
the program year, and the period in which active reform occurred. Note that since students take, 
on average, nine to ten semesters to graduate but the degree program is officially eight semesters, 
students get out of synch with each other. Thus some courses are listed in two different program 
years. Two of the courses that were initially slated for reform did not get reformed, represented 
by “NA” in the table below. The course Engineering Your Future encountered numerous obstacles, 
and despite significant efforts and investment the course never converged into a teachable format. 
This was a significant setback, and many of the elements originally scheduled for this course were 
later implemented in Applying Mathematics to Engineering Design. Networks and Linear Systems 
was almost fully developed when the faculty member in charge of the course moved to another 
university. His untenured replacement did not feel that course reform was in his best interests in 
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Program Year 

Reform Period 

Course and Description 

Fr. 

2007-2010 

Applying Mathematics to Engineering Design: Teaches how mathematics is 
used in the engineering design cycle (M) 

Fr. 

NA 

Engineering Your Future: Teaches students to prepare portfolios, how to 
monitor their development, and teach skills in teamwork and peer evaluation. 

Fr. 

2010 

Introduction to Engineering: Introductory course mainly focusing on 
retention, but also has elements of ethics and design. 

So. 

2005 

Electrical Science: Basic circuits course for all engineers (S). 

So. 

2010 

Experimental Methods: Teaches basic electronic measurement techniques 
and principles of system design (S). 

So./Jr. 

2004 

Introduction to Digital Logic Design discrete and field programmable gate 
array logic devices (S). 

So./Jr. 

2006-2008 

Semiconductor Device Physics for Engineers: Physics of semiconductor 
devices (M). 

So./Jr. 

2007-2009 

Microcomputer Architecture: Basic architecture, I/O. and programming of 
embedded microcontrollers (S). 

Jr. 

2003-2006 

Electromagnetic Fields: The standard EM fields course covering electro- and 
magneto-statics and plane waves (M). 

Jr. 

NA 

Networks and Linear Systems: This course teaches the basics of linear system 
theory relying heavily on application of mathematics (M,S). 

Jr./Sr. 

2003 

Engineering Optics*: Optical system design for engineers (S) 

Sr. 

2005-2007 

Laser Electronics*: Course focusing on laser design and modeling as an 
example of complex, tightly coupled systems (S). 

Sr. 

2007 

Microwave Systems*: Design of high frequency devices and systems. 

Students design and build a synthetic aperture radar (S). 

Sr. 

2005-2010 

Design of Engineering Systems: Teaches engineering design from a systems 
perspective through team projects (S). 

Sr. 

2006-2008 

Senior Design: Teams of students undertake independent design projects 
under guidance of a faculty mentor (S). 


Table 1: Engineering Students for the 21st Century Courses. 


the first years of his career. As the reform project winds down, two new courses are under develop¬ 
ment, an introductory course in the freshman year and an experimental methods course taken in 
the sophomore year. Although not anticipated in the initial planning, three elective courses (*) are 
also incorporating the ES21C format. 

The strategic approach planned for ES21C courses was to first structure learning around two or 
three fundamental problems or projects rather than a list of topics or needed content. To pose the 
fundamental problems in a relevant context, each project begins with a case study or other means 
of contextualizing course material. Preparing a conceptual foundation and teaching students the 
process of solving the problem occurs in three parallel steps based on the cognitive process di¬ 
mension of Bloom’s Taxonomy (Krathwohl 2002): (1) remember and understand concepts needed 
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in solving the problem outside of the classroom; (2) during class student teams apply what they 
know and analyze the problem using active learning; (3) outside class (typically in a lab) student 
teams create a solution to the problem through a design project then evaluate how well it works. 
To support teamwork, ES21C developed resources for team building and peer feedback; these are 
discussed later. At the end of each project, teams communicate what they have learned, ideally by 
creating an engineering portfolio. Each class was also supposed to integrate reflective activities to 
help students learn from their experiences, both good and bad. These features (strategies) guided 
the curriculum reform strategy, but individual faculty were free to choose techniques (tactics) that 
best suit their teaching style, course goals, and the level of their students. 

Educational Basis of Engineering Students for the 21 st Century 

The Engineering Students for the 21 st Century reform vision outlined above seems simple at first 
glance, but has at its core several fundamental beliefs that run counter to current practice and the 
beliefs of some faculty: 

• Contextualizing learning around projects implies that learning goals are determined by the 
project as much or more than some external set of facts, concepts or topics such as those 
found in a textbook. If some subset of factual, conceptual, or procedural knowledge is not 
needed to complete the project it need not be taught. 

• ES21C recognizes that learning is developed over time, is based on an individual’s prior 
knowledge and experience, and has hysteresis; reform tries to create a “knowledge-centered 
environment” (National Research Council 2000) in courses. Thus different students may draw 
different lessons from a class; in ES21C this is expected and desirable. 

• Since development is incremental and individual, students’ paths may diverge throughout their 
degree program as they develop into creative and innovative engineers. Graduates should be 
prepared to go on to diverse careers, even if these are not in engineering. 

Given ES21C’s non-traditional assumptions, this section briefly outlines the basis in learning theory 
for key elements of ES21C courses. 

Structuring courses around problems is drawn from Problem-Based Learning (PBL) (Barrows 
and Tamblyn 1980; Woods 1996). In ES21C, PBL takes place in three steps to teach the process 
of problem solving. First, course work is contextualized around the problem allowing students 
to determine if what they learn applies to their project, developing metacognition (National Re¬ 
search Council 2000). Second, students implement a solution to the problem as part of a team; 
implementation presents students with ill-defined problems (PBL website 2002, National Re¬ 
search Council 2000) that support transfer of learning (Bjork 1994). Third, students evaluate then 
report on their team’s implementation, often by creating a datasheet. Quantifying the project’s 
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performance helps students see the utility of course content. These steps are described in more 
detail in the following paragraphs. 

The first step, creating context, allows students to make connections between what they know, 
transfer that knowledge to new situations, and continually practice these skills- hallmarks of deep 
learning. Such interconnections and transfer occur more easily if a student has a framework or “scaf¬ 
fold” on which to attach new knowledge (National Research Council 2000). All students have such 
scaffolds that are built from prior experiences; since all students are unique individuals with different 
experiences, each scaffold is different. It is virtually impossible for faculty to know students’ precon¬ 
ceptions, and difficult to correct misconceptions that occur when students learn wrong information 
or transfer knowledge to an inappropriate context (Burbules and Linn 1992). 

ES21C courses create scaffolds by introducing projects through a case study. Case studies 
provide intent (Minniger 1984) (i.e. the student knows the knowledge will be used), makes knowl¬ 
edge relevant (McCombs 1996; Pintrich and Schunk 1996), and places work in a context related 
to students’ prior experiences. Placing knowledge in context (Glaser 1992) enhances transfer 
(National Research Council 2000; Gick and Holyoak 1983). The case study explicitly asks students 
to self-identify personal learning goals for the course. In-class discussion of the case study also 
helps define the problem in terms of familiar schemas (McVee, et al. 2005); being able to think in 
terms of schemas is one characteristic of an expert. For example, an electrical engineer can look 
at a large circuit schematic and identify groups of individual components as functional elements, 
dramatically simplifying overall conception of the circuit’s function. Since students in a class do 
not have the same schemas, in-class discussion of the case study allows students to recognize dif¬ 
ferences between their individual schemas, develop a group schema for the project, and identify 
areas in which they have low proficiency. Case studies let students utilize experiential or narrative 
(Epstein 1994) modes of processing information rather than the academic language so often used 
in traditional courses (Gee 2004). Using a case study before teaching concepts improves learn¬ 
ing more than using it afterwards (Lundeberg and Schuerman 1997). In the planning phase of the 
project students’ self-reports of learning gains showed that the case study was rated as having the 
largest positive impact on learning. 

The second step in ES21C’s implementation of PBL is to have students solve the problem posed in 
the case study. Contextualizing classes around a project requires that the classroom experience be 
transformed to allow the instructor to focus on the process of solving problems rather than transmitting 
knowledge. ES21C accomplishes this by first classifying course content using the taxonomy described 
in the next section. Students acquire information (understand cognitive process level) outside the 
classroom, address apply and analyze processes in the classroom, and develop design skills through 
building the project. Thus, while traditional courses transmit content in class and assign problems to 
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be done outside of class, in ES21C this model is flipped and information is acquired outside of class 
and problems are done in class. To facilitate learning, the instructor needs to provide resources that 
help students acquire information; examples include short videos (Cheville 2009d, hyperlink ), web 
sites, and readings from the textbook. In putting the onus of knowledge acquisition on students it is 
important to provide formative evaluation—feedback during learning—quickly enough so that students 
can correct their misconceptions and develop metacognition (Ericsson, et al. 1993). New technologies 
such as course management systems and on-line Java applets support such formative evaluation. 

While students can learn lower cognitive process levels of the taxonomy outside of the classroom, 
higher levels are addressed by active learning in class, often using teams. Active learning simply 
means students are active participants in the class rather than passive listeners (Smith et al. 2005). 
A significant body of research has shown that in well-organized classes active learning gives rise to 
significant improvements in performance (Schwartz et al. 1999; Froyd 2007). Active learning helps 
change students’ perception that the instructor is responsible for learning. In ES21C, active learn¬ 
ing is framed in the context of the problem addressed in the case study and project and provides 
students information needed to complete their project. The instructor should inform students of 
how learned knowledge applies to the project and be integrated with learning from previous classes 
to develop near transfer (Alkon 1982; Druckman and Bjork 1991). Transfer—the ability to take what 
has been learned in one context, and extend it to a different context—is a key ability in “problem 
solving” (Jonassen 2000), a skill that faculty in our program value. Active learning connects the 
material read outside of class to the project that scaffolds the class. 

The project is done outside of the class time, typically in a laboratory environment, although some 
instructors have used take-home kits. In some ES21C classes parts of the project are done as active 
learning in class and teams simply need to integrate work they have already performed in completing 
the project. A project should be involved enough that it is too complex for one individual and extend 
over several weeks at a minimum (Smith et al. 2005). Teams, when properly implemented, improve 
learning outcomes (Smith et al. 2005) and allow more authentic (and interesting!) problems to be 
assigned. Having students work in cooperative teams has been shown to benefit non-traditional 
students and women (Seymour and Hewitt 1994). ES21C has also developed a peer feedback system 
to help students learn from team experiences; this is discussed subsequently. 

The third step in ES21C’s implementation of PBL is for students to report on the results of the 
team project. Reports include both team and individual contributions; individual writing assignments 
ask students to self-explain (Chi et al. 1994) and reflect on the learning experience. A large fraction 
of students’ grade (50% in some courses) is based on communicating the results of technical work 
to help ensure students take reporting seriously. 
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CHANGES ENGENDERED 


The previous sections described the vision, history, and strategy of Engineering Students for the 
21 st Century , and discussed the project’s basis in learning theory. Since ES21C seeks reform at the 
department level, the project is also changing the learning environment. This section first outlines 
ongoing changes to the learning environment with an emphasis on tools that can be adopted by 
others. Next, four short case studies illustrate how the ES21C model and changes to the learning 
environment reformed select classes. 

ES21C has engendered several substantial changes and improvements to the resources available 
to faculty across the program to support learning. As discussed previously, a key element of ES21C 
is classifying learning using a taxonomy to help determine what aspects can be pursued indepen¬ 
dently by students outside of class, and which should be addressed by projects or active learning in 
class. Initially ES21C used Bloom’s Taxonomy, but this was quickly found to be too general for most 
faculty to use successfully without a significant time investment (Abe and Starr, 2004). To better 
understand and classify learning, an engineering taxonomy was developed. Bloom’s Taxonomy has 
previously been adapted to measure ABET outcomes in engineering programs by addressing each 
of ABET’s (a)-(k) outcomes (ABET 2008) on seven cognitive process levels (Besterfield-Sacre et 
al. 2000). While this taxonomy directly assigns levels of ability to the requisite ABET outcomes, 
comprehensiveness comes at the cost of complexity. In creating an engineering taxonomy for ES21C, 
the cognitive process dimension of Bloom’s Taxonomy was reduced from six to four elements. On 
the knowledge dimension factual and conceptual dimensions were retained unchanged and the 
process dimension included five aspects corresponding to sequential steps in the engineering design 
process developed for ES21C: researching, modeling, implementing, measuring, and communicating 
(see Figure 3). The expansion of the process dimension reflects the importance ES21C places on 
design (Sheppard et al. 2008). A short version of the design taxonomy focusing only on process 
skills is shown in Figure 2; a complete version can be downloaded from the ES21C website (Cheville 
2009a, hyperlink ). Compared to design cycles published in texts on design (Ford and Coulston 
2007) ES21C developed a simplified cycle with fewer steps, Figure 3, to simplify classification of 
course activities and eliminate design process steps that posit high-level thinking skills which could 
invalidate the taxonomy. 

To better understand students’ learning experiences, a survey based on the engineering tax¬ 
onomy was developed and given to faculty in the ECE department in 2007 (Cheville, et al. 2008). 
For each element of the taxonomy (cell on the table in Figure 2), the survey asked faculty to 
report how much impact work of that type (i.e. understand modeling) had on the overall course 
grade. Faculty members were also asked to rate the relative importance of each element of the 
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Cognitive Process Dimension 



Understanding: being able 
to recall what you learn and 
define, explain or give 
examples to others. 

Applying: using information 
to accomplish something 
concrete. 

Analyzing: being able to 
extract meaning from a 
collection of information. 

Designing: creating 
something new under 
constraints to achieve a 
near optimal outcome. 

Research 

Learning about a 
problem before you 
solve it. 

Explain a problem in your own 
words, find information 
needed to solve the problem, 
and determine what you need 
to know to solve the problem 

Fnd and use information 
needed to solve a problem 
from textbooks, on-line 
resources, or other technical 
sources. 

Select and filter your 
results to find information 
that is useful in solving a 
problem. 

Use the results of research 
to select a productive 
approach and implement a 
specific plan of action. 

Model 

Using mathematics 
to understand and 
make predictions 
about system 
performance. 

Solve and graph single 
equation models and identify 
the conditions under which the 
solution applies 

Compute or solve a 
mathematical model of a 
physical system and plot the 
behavior of the system either 
analytically or 
computationally. 

Perform calculations 
which predict performance 
of a device or system and 
test/validate these 
predictions. 

Optimize the performance 
of your model and use an 
engineering model to make 
design decisions. 

Implement 

Going from ideas, 
models, or 
concepts to a 
physical realization 
of your design. 

Explain how to turn the design 
into an actual physical object 
if you are given standard 
engineering drawings or 
diagrams. 

Build a simple engineering 
device, follow a process, or 
write code using appropriate 
tools and techniques 

Select between different 
fabrication techniques and 
select appropriate parts or 
materials given project 
constraints. 

Construct a device with 
multiple, interconnected 
systems in an efficient, 
affordable way that meets 
external constraints. 

Test 

Determining how 
well a system or 
device functions by 
performing 
measurements. 

Identify the correct 
measurement instruments for 
a task and write down the 
procedure for taking a 
measurement correctly. 

Take valid measurements 
using instruments and record 
the data in a format that can 
be shared with others. 

Use data you to determine 
if a device is functioning to 
a given specification ora 
process meets external 
requirements. 

Choose or create correct 
measurement techniques, 
acquire interpret data, and 
then use this information to 
make design decisions or 
determine if a device meets 
specifications. 

Communicate 
Expressing results 
of work so others 
can understand, 
reproduce, and 
improve it. 

Identify appropriate formats 
and styles for your technical 
communications as well as 
what information will be 
communicated for each 
format. 

Present evidence to an 
audience at a level they can 
understand either in writing, 
through oral presentations, or 
at a meeting. 

Edit your own work or the 
work of others for style and 
content to meet the needs 
of a particular audience. 

Create a well-reasoned, 
data-driven argument or 
presentation either from 
your own work or from 
work that others have 
done. 


Figure 2. The process portion of the ES21C engineering taxonomy. The complete 
taxonomy, guides on using the taxonomy, and support software can be found at this 
hyperlink 


cognitive process and knowledge dimensions to their course outcomes on a five element Likert 
scale. “Researching” was the aspect least reflected in student work, particularly in the sophomore 
and junior years. Faculty also reported that “communicating” had little impact on grades despite 
the fact faculty reported this as an important skill. Faculty reported students performed the most 
work on conceptual knowledge and modeling/calculating. The development of an engineering 
taxonomy has given ES21C a means both of classifying student learning to determine which activi¬ 
ties are appropriate for different pedagogies, but also a way to measure changes in what students 
learn over time. A complete description of the Engineering Taxonomy and survey development 
has been submitted for publication to Advances in Engineering Education. 

Another tool developed under ES21C was a flexible method of supporting peer evaluation in 
team experiences. As mentioned previously, team work is an integral element of the ES21C model. 
During the project a web-based peer evaluation system was developed both to collect data on team 
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Multiple Iterations 

The Engineering Design Cycle 

Figure 3. The ES21C engineering design cycle the program is using to frame design 
classes and which define the process elements on the engineering taxonomy (Figure 2). 


performance and provide feedback to students on their performance on the team. While other peer 
evaluations exist (Ohland 2009), the ES21C-developed system—the Simple Team Experience As¬ 
sessment Measure , or STEAM—is highly configurable by the instructor to meet the specific needs 
of their class. Five different evaluation tools can be fully configured by the instructor: a Likert (1-5) 
scale rating of team member attitude and value, numeric reporting by peers on the perceived work 
put forth by team members, a survey designed to measure a team member’s effectiveness, anony¬ 
mous text feedback, and an overall rating. Almost any combination of tools may be configured 
depending on the needs of the class. STEAM has been used in several classes to evaluate changes 
to student learning and their effectiveness on teams; for more information consult (Cheville, et al. 
2007; Cheville and Duvall 2008) or to use STEAM visit the peer evaluation web site (Cheville and 
Duvall 2009, hyperlink to STEAM site ). An on-line electronic contract that allows students to specify 
their own roles and tasks and then be evaluated on meeting the terms of the contract at the end of 
the project is in beta test phase; if interested please contact the lead author. 

Engineering Students for the 2T‘ Century is also changing the departmental infrastructure to 
better support projects and empower engineering students to explore all aspects of engineering 
design by slowly transitioning away from the model of a pre-configured lab bench. Using commer¬ 
cially available electronic catalog software, students or teams can check out instrumentation from a 
department web site to use over the course of the semester. By checking out equipment, students 
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are responsible for the equipment and can access lab facilities on a 24/7 basis. Three different 
catalog systems (Cheville 2009a) support different aspects of learning: one is to check out equipment 
that will be returned, a second lets students purchase expendables such as electronic components, 
and the third is used to sell kits developed under ES21C. To teach the actual cost of engineering 
projects, electronic equipment is priced at actual cost, however the equipment is loaned to students 
rather than purchased. The catalog helps develop student discernment in lab equipment selection 
without the need to teach extra content on laboratory instrumentation since students must exercise 
judgment in choosing appropriate instrumentation, and also build their own test and measurement 
systems to meet team needs. The rate of broken and damaged equipment has been reduced since 
students take better care of equipment they are personally responsible for. 

Since the challenges and triumphs of an undertaking the size of department-level reform are 
impossible to fully document in a single article, an overview of how ES21C was implemented in four 
courses follows. These short case studies of successfully reformed classes cover the spectrum from 
freshman to senior, theory to application, and purely electrical engineering to multi-disciplinary 
service course. Each case outlines the changes that were made to the course, and briefly reports 
evaluation results. Where available, references are made to publications that present more details. 

Case Study #1: Electric Circuits (ENSC2613) 

Like many engineering schools, a wide range of students take a requisite engineering science 
course in electric circuits, part of the pre-engineering curriculum. Classes are large, typically 
around 200 students. While for electrical engineers the course is a critical prerequisite and taken 
in the sophomore year, students in other majors may delay taking the course until they are seniors. 
ENSC2613 was one of the first courses reformed under ES21C; the course was improved over several 
iterations and has currently reached a steady state. 

Before the changes made under ES21C, the course was taught in a traditional format with several 
lectures per week and regular summative homework assignments and tests. Since course content 
is determined by an interdisciplinary faculty committee, there were few changes to the content. To 
show the impact of ES21C, the instructor chose to adopt an ABAB model where “A” corresponds 
to a nominally four week period of lecture and summative assessment and “B” to four weeks of the 
ES21C model outlined previously. The only changes to the ES21C model outlined previously were 
reducing the scope of the projects, giving teams a written case study rather than active discussion 
in class, and changing the format of in-class active learning. Due to the large size and diversity of 
the class in-depth projects were not feasible, so the instructor chose to implement a series of shorter 
team projects. Projects were introduced through a short case study that discussed a problem at an 
engineering firm. In-class active learning consisted both of traditional written problems as well as 
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measurement challenges using a project kit provided to each team; kits contained a protoboard, 
components, batteries, and digital multimeter. Selected teams demonstrated their solutions in class 
on a computer with data acquisition and virtual instruments running in LabVIEW® so results could 
be shared with the class. 

Initially the ABAB structure was done as part of a research study to assess the impact of ES21C on 
student learning. After the end of the study the instructor chose to continue the ABAB model since 
he believed students with different learning styles benefitted from experiencing different pedago¬ 
gies; the accuracy of this belief has not been verified experimentally. The instructor reported there 
was initially considerable “push-back” from students who felt they were not being “taught” since 
the role of the instructor is less active under ES21C. This attitude has decreased with subsequent 
iterations as ways have been found to inform students that the course closely mimics engineering 
practice. Over time, the instructor has made increasing use of the engineering taxonomy (Cheville, 
et al. 2008) to ensure out-of-class work addresses appropriate cognitive process levels. 

Learning in ENSC2613 is being evaluated using a Student Assessment of Learning Gains (SALG) 
(SALG Website 2009), summative case analysis tests (Lundeberg and Schuerman 1997), and short 
assessment questions posed as part of each project report. Initial analysis of SALG results show 
there is quite a wide range of student opinion about the course; some students report they are 
not being taught while others embrace the more proactive approach to learning. The case analysis 
tests, done in a pre-post format for each part of the ABAB research design, show greater learning 
gains on the ES21C-taught B units, but also significantly larger standard deviation. This is consistent 
with SALG results if there is a group of students who disengage from the course. The instructor, 
who has significant prior experience teaching using the ES12C format, has reported differences 
in the pace at which students learn in the A vs. the B units. In the more traditional A approach 
there is little apparent effort by students until just before homework is due at which time there 
is a large spike in student effort. Under the ES21C model, a more moderate pace of work occurs 
throughout the entire week. Overall the instructor reports “enrichment” of the learning environ¬ 
ment with less opportunities for students to “slip under water”, i.e. drop out of the course with 
little prior notice. The instructor’s awareness of student status increased significantly after peer 
evaluations were introduced. A more complete description of changes to learning can be found 
in (Yadav, et al., 2011). 

Case Study #2: Design of Engineering Systems (ECEN4013) 

As with most engineering programs, ECE students undertake a capstone design experience 
in their final year; ECE requires two sequential capstone courses. For many students the cap¬ 
stone course fit Sigmund Freud’s definition of experience: “Experience is what you get when you 
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experience what you do not wish to experience." The first course, described here, prepares students 
for independent, team-based design projects in the second course. Prior to the start of ES21C, the 
first capstone course consisted of a series of tightly constrained electronic design projects performed 
by individual students. Grades were determined by quantitative performance metrics (i.e. signal to 
noise ratio, accuracy of gain, etc.). A survey of students who had completed the course showed 
students felt fundamentally unprepared for the design experience. A representative comment was 
“...all of the undergraduate level courses did not build up to this course. This was like ‘wanting to fly 
without even knowing how to crawl”’. 

Early in the course reform process, faculty realized that the course was too focused on completing 
projects without having identified clear and teachable learning goals, resulting in a highly variable 
rate of project success. The ES21C model provided guidance on effective pedagogy while learning 
goals were identified. Key elements drawn from ES21C were the focus on projects, communication, 
active learning, and working in teams. Action research (Bodner and Orgill 2007) was then used to 
develop and implement learning goals in engineering design that were relevant to students. The 
instructor first set initial learning objectives, engaged stakeholders—including students, alumni, fac¬ 
ulty, and the advisory board—in identifying most and least effective practices, and then iteratively 
made changes to the course. From this process several key learning objectives emerged: teaching 
design as a process, ensuring students gain experience at each process step, teaching elements of 
project management, and ensuring each student had a meaningful role on design teams that included 
individual accountability (Cheville, et al. 2007; Cheville and Duvall 2008; Co, et al. 2007). 

Even more than other ES21C courses, the project forms the central core of ECEN4013, contextual¬ 
izing all learning. Students meet in class weekly to learn project management skills through active 
learning. The majority of student time is spent in the lab which consists of a five week training project, 
a nine week design project, and a week of reflection and reporting. The training project teaches ele¬ 
ments of electronic device fabrication that are not covered earlier in the curriculum, ensuring each 
team member can contribute meaningfully to projects. Students learn fabrication, programming, or 
test and measurement skills and then apply these skills to fabricate a working electronic device from 
a schematic diagram as part of a team. The electronic peer evaluation instrument discussed previ¬ 
ously (Cheville and Duvall 2009) is used to accelerate team development (Smith et al. 2005). The 
design project follows the five step design process developed as part of ES21C (Figure 3). At each 
step students must demonstrate a minimum level of competence to advance to the next stage of 
the design cycle. The last week of the course is devoted to reflection and reporting to give students 
experience in technical communication and reflect on how the experience was transformative. 

Since the capstone course sequence is critical to ABET program evaluation, a significant effort 
has been made to assess course outcomes. Both qualitative and quantitative metrics paint a picture 
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of students gaining a realistic perspective on the social process of engineering design, including the 
uncertainty of the process, importance of broad technical expertise, and need for negotiation and 
compromise (Bucciarelli 1994). It is clear from qualitative evaluation of reflective statements that 
students’ experiences vary greatly depending on the roles they take on and their level of engagement. 
The observation that learning is dependent on a student’s goals reflects one of the fundamental as¬ 
sumptions underlying ES21C’s overarching goals as discussed previously on the section on ES21C’s 
basis in learning theory. More details on the specific implementation, theoretical foundation, and 
evaluation of this course can be found in (Cheville 2010). 

Case Study #3: Semiconductor Device Physics (PHYS3233) 

A required course taken late in the sophomore year or early in the junior year, PHYS3323 has a 
convoluted history. Well before the start of ES21C students took engineering science courses on 
fluid dynamics and strength of materials. Since ECE faculty felt these courses did not support other 
courses in the curriculum, a survey course in modern physics was substituted. Later, program evalu¬ 
ation showed that the modern physics course content varied greatly depending on the instructor, 
and that students thought the content was not germane to electrical engineering. Faculty in ECE 
then worked with colleagues in Physics to develop a course in semiconductor device physics that 
covered the basics of modern and solid-state physics, semiconductor materials, and the PN junc¬ 
tion. A veteran instructor taught the course in a traditional lecture-homework-summative exam 
format for several years before the start of ES21C. The instructor was very interested in course 
reform since he reported that while students made acceptable grades on procedural problems, 
they lacked conceptual understanding. 

Attempts to reform the course at the beginning of the project faltered since the pedagogies 
called for in ES21C were unfamiliar to the instructor and TA’s in the Physics department. As the 
project progressed, a team consisting of graduate students and faculty in ECE and Physics made 
a second, successful attempt at course reform. In the spring semester of 2008 a trial run at a re¬ 
formed course model was made in a ABA format; approximately five weeks of an ES21C model was 
taught first (A), followed by traditional lecture (B), and concluding the course with the ES21C model. 
This preliminary implementation gave only anecdotal evidence of a positive impact on learning. 
Students self-reported better understanding of topics and performed better on problems on the 
portions of the final summative examination that were taught using the ES21C model compared to 
the traditional lecture style. 

At the end of the spring semester the original instructor announced he would retire, and a new 
physics faculty member agreed to teach the course in Fall, 2008. The new instructor was initially 
highly skeptical that the active learning approach used in ES21C would allow students to learn the 
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highly conceptual and emergent (Chi 2005) material covered in this course. Thus in the summer of 
2008 the two physics faculty and the PI of ES21C worked with graduate students to complete the 
course reform and address the new instructor’s concerns. The content was reorganized into seven 
modular units, each organized around an engineering-related context. The course begins by ex¬ 
plaining how night vision goggles work by introducing uncertainty and the photoelectric effect and 
ends with PN junction diodes. Each unit follows a three-step timeline. The first day reviews learning 
objectives and outlines major concepts of the unit using lecture. In the middle portion of the unit 
readings drawn from the textbook, web sites with illustrative JAVA applets, and other sources are 
assigned before class. To ensure students have a basic understanding of course material, a formative 
on-line quiz is given over each reading. To prepare students for active learning exercises done as 
team in class, they are given the in-class exercise (ICE) before class with all numeric data blacked 
out. At the start of class each student turns in a written strategy for solving the ICE in the form of 
pseudocode or a flowchart. This prequel familiarizes students with the problem and helps ensure 
students do not spend class time familiarizing themselves with the problem. The complete ICE is 
then solved by teams of students in class. The instructor circulates throughout the room assisting 
students or giving short explanatory lectures as needed. The final day of the unit serves as a review 
and wrap-up. The formative quiz for the review day consists of conceptual, multiple choice ques¬ 
tions reviewing module topics and an on-line forum where students submit questions; questions are 
reviewed and graded by the instructor. In the classroom the instructor answers student-submitted 
questions and corrects misconceptions about the unit. The one exception to this format is the “crystal 
zoo” module which, due to content, is taught using traditional lecture. 

Content was developed at weekly meetings in summer 2008 that involved the three faculty and 
graduate students. Over the first month progress was slow as participants became familiar with 
ES21C pedagogy and how it was applied in the classroom. Over time, each faculty member took 
on the role of an expert in some aspect of course development. The new physics instructor had 
clear learning and content goals and a deep understanding of device physics, his research area. The 
outgoing physics instructor had years of experience with students’ conceptual difficulties and prior 
preparation and knowledge. The engineering faculty member provided expertise on pedagogical 
techniques and was able to contextualize content so it was relevant to engineering students. Once 
roles were established, course development proceeded rapidly through the summer and at bi-weekly 
meetings during the fall semester. 

As mentioned previously, the instructor who took over the course in fall 2008 was deeply skeptical 
that the active learning approach would be effective for the conceptually difficult material covered 
in PHYS3233. Over the course of the semester his attitude underwent a 180° shift. Despite some 
disengaged students, overall both teams and individual students performed well on assignments 
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and exams. Class attendance and engagement were higher on active learning than lecture days and 
the instructor received a great deal of feedback on student learning at the end of module question 
session and by monitoring team progress during ICEs. Overall course grades were quite high, with 
non-engaged students doing significantly more poorly than peers. These experiences mirrored those 
of the previous physics instructor in the Spring 2008 trial. Students self-reported high learning gains 
and high satisfaction in the end-of-semester SALG (SALG Website 2009) that was developed for 
the course. 

Case Study #4: Engineering Mathematics with Design (ENGR1113) 

OSU students take a two year “pre-professional” course sequence common to all engineering 
students before admission into one of six engineering professional schools (departments). From 
a logistics viewpoint, the first two years of the program turned out to be more difficult to reform 
since change impacts students outside the electrical and computer engineering department and 
thus requires consent from a large number of stakeholders. 

ENGR1T13 is a three credit hour course (standard at OSU) taken in the first semester of the fresh¬ 
man year and reform was undertaken fairly recently. The three credit hour course was created 
by removing two introductory engineering courses; ENGR1113 retained critical elements of these 
courses. The course introduces students to engineering through design projects in which they ap¬ 
ply mathematics and learn the five-step ES21C design cycle, Figure 3. ENGR1113 is loosely modeled 
on previous efforts at Wright State University (Klingbeil 2005) to improve retention by illustrating 
mathematics in engineering, but additionally introduces design using the ES21C design cycle. The 
course follows the ES21C model by integrating case studies, projects, written communication, read¬ 
ing outside of class, formative on-line quizzes, and having students work in teams. 

The class meets three times per week. Projects are introduced on Thursdays and all projects are 
contextualized around real-worid applications. For example, a project illustrating linear algebra 
discussed mixing syrups and carbonated water for soft drinks as a chemical engineering process. 
Reading and formative on-line quizzes are assigned on a weekly basis and a recitation session 
helps students who need assistance understanding concepts. The on-line quiz prepares students 
for the Wednesday lab in which teams of three students take on six challenges throughout the 
course; each challenge has both engineering analysis and engineering design components, and 
each is contextualized around a real world problem. The six challenges have students apply alge¬ 
bra, linear algebra, trigonometry, sinusoids (waves), derivatives, and integrals. Lectures are given 
once a week to introduce new topics and illustrate application of mathematics to the engineer¬ 
ing problem. Each lecture solves problems twice, once using an analytic approach then revisiting 
the problem and solving it numerically using Matlab. To report on their work, teams submit three 


SUMMER 2011 


21 






ADVANCES IN ENGINEERING EDUCATION 

Engineering Students for the 21 st Century: Student 
Development Through the Curriculum 


written reports over the semester that are reviewed by peer writing coaches before final submis¬ 
sion to allow feedback on writing performance. Individual students write reflective statements over 
topics in engineering, mathematics, and design; both reports and statements are graded using a 
rubric given to students. 

Since ENGR1113 has only been offered once, evaluation of course outcomes are still preliminary. A 
pre-post attitude survey and pre-post mathematics concept inventory were given using a one hour 
introduction to engineering course as a control group (Oswald, et al., 2009). For the concept inventory 
both sample (N = 37) and control (N = 32) groups performed better on the post test, but the difference 
was not significant. The control group did significantly better on both pre- and post- tests than the 
sample group; the difference is likely due to self-selection since students who felt they needed help 
in mathematics were encouraged to enroll in ENGR1113 by counselors. On the mathematics portion of 
the ACT test, ENGR1113 students performed 1.4 points lower than the control group; the difference was 
not significant at p < 0.2. ENGR1113 students scored 1.2 points higher, however, on both reading and 
science reasoning (p < 0.25) The mathematics attitude survey asked about intrinsic motivation, pre¬ 
conceived notions about mathematics and engineering, and motivation to continue in mathematics. 
Analysis of variance showed that for all categories the control group had a significantly lower mean 
(p < 0.05) than the sample, indicating that the new course had a positive impact. 


CHANGES IN STUDENT LEARNING 


The goal of Engineering Students for the 21st Century’s reform of department infrastructure and 
courses was to improve student learning. To understand changes to student learning, ES21C initially 
outlined four research questions focused on whether students learned “better” in reformed courses. As 
the project has evolved it has become increasingly clear that while the question “Are students learning 
better?” is often asked, the answer is almost always “It depends”. A better question is “What are students 
learning?”, which requires faculty to first clearly define what they want students to learn. Faculty often 
overlook the importance of identifying achievable learning goals, set goals that are not measureable, or 
focus exclusively on content to the exclusion of procedural skills and metacognition. Through ES21C the 
program has made progress identifying learning goals across the curriculum by working with faculty to 
write “Be Able To’s” that define measurable outcomes for each class; this work is still ongoing. While 
much of the evaluation done under ES21C to date has been focused on courses, a holistic view of how 
ES21C has changed graduates' experiences is slowly emerging. This section discusses the progress that 
has been made to date in changing student learning and attitudes program-wide. 
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A variety of metrics are used to monitor student learning. Broad trends in student learning are 
measured through both quantitative and qualitative measures. To be able to sustain reform at the 
end of the funded phase of the project, these measures have been closely aligned with program and 
learning evaluation that is performed as part of the ABET CQI cycle, as will be discussed later. 

One measure used to determine the impact of ES21C on students is a phone survey of undergradu¬ 
ate alumni who graduated one year and five years ago that is conducted by OSU every other year. 
While most survey questions have Likert-scale (1 = low, and 5 = high) responses, some open-ended 
questions are also asked. Survey responses from 2002 to 2008 (the last year data is available) 
were analyzed to determine changes in alumni’s view of the program over the period ES21C was 
implemented. Results for two groups of questions are shown in Figure 4, the satisfaction students 
have with the program, and preparation in design and teamwork and communication. The five year 
alumni data are shown in blue while the one year alumni data are shown in red. Note that there are 
one and five year lags between survey data and when effects would be observed. Dashed arrows 
represent points in time alumni would first have experienced changes due to the initial planning 
phase of ES21C that started in 2003 and solid arrows represent full implementation in 2005. The five 
year alumni figures (shown in blue) generally lag those of the one year alumni (shown in red) which 
provides supporting evidence the changes are due to programmatic changes rather than changes 
to the survey methodology. The response rate varied between approximately 20% and 40% for the 
five year data, and 40% and 60% for the one year data, leading to sample sizes from N = 19 to N = 
64 depending on the year the data was taken. 
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Figure 4. Responses to an alumni survey for alumni one year (red) and five years (blue) 
following graduation. The dashed arrows represents the first point in time alumni would be 
impacted by the planning phase of the project, and the solid arrows the full implementation, 
(a) shows design and teamwork & communication skills, while (b) is overall satisfaction with 
the program. 
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The data shown in Figure 4 is supported by qualitative, open-ended survey question responses. 
The number of alumni who cite design as a strength of the program in open-ended comments 
roughly doubled from 2002-2004 (pre ES21C) to 2006-2008 (post ES21C) from 20% to 40%. Note 
that there is no way to separate one-year and five-year graduates on open-ended responses, but 
the number of one-year respondents is larger. A similar magnitude drop was observed in the num¬ 
ber of alumni who reported lack of design opportunities was a weakness. While not as dramatic, 
increases in alumni who report teamwork and communication as both a strength and a weakness 
of the program have increased. It is not clear why more alumni report that the program is getting 
better and needs to get better still. It may be that more alumni are aware of the importance of com¬ 
munication and teamwork skills in their engineering careers. While responses on design preparation 
and overall satisfaction with the program increased, there were not significant changes in alumni 
reports of preparation for graduate school, professional and community involvement, or prepara¬ 
tion in social and economic aspects of engineering. The survey data provides indirect support that 
ES21C is having intended changes to program graduates. While other factors cannot be ruled out, 
there were no other major changes to the degree program during this time period. 

To measure changes over time in student attitudes and beliefs, an exit survey is given to students 
just prior to graduation. All students are required to complete this survey leading to sample sizes 
(depending on semester and year) from N = 12 to N = 54. A group of questions on the survey asks 
students to self-report their perceived ability in multiple areas. While self-reporting of knowledge or 
ability may be suspect as objective data (Kruger and Dunning 1999), survey results are interpreted 
as relative changes in the perceived ability of student cohorts over time. Survey questions were 
grouped into five general categories: coursework, breadth, design, professional development, and 
context. The coursework category included seven questions covering students’ ability in different 
courses required to earn a degree. This category was used as a control, since the questions focused 
on courses that did not participate in ES21C. There were four questions in the breadth category that 
focused on the extent or range of student abilities in experiments, systems, analysis, and design. 
The design category asked four questions on teamwork, communication, and perceived ability on 
design projects. Six questions in the professional development category focused on professional 
development and activities. The context category asked four questions about students’ ability to 
place engineering work in societal, environmental, global, and economic contexts. 

Seven years of survey results are shown in Figure 5(a); data was not collected the fall semester 
of 2003. The data has been normalized to the coursework responses and shows changes in the 
mean response over time. From the start of ES21C implementation in Fall 2005, students perceived 
they had a wider breadth of ability, which is expected based on the fact that ES21C attempted to 
expand the domains of engineering knowledge taught in the program. Students’ self-perceived 
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ability to contextualize engineering increased slightly, but gains in professional development, a 
goal of ES21C, did not occur. Surprisingly, was there little increase in self-perceived ability in de¬ 
sign. When the design category was separated into components of teamwork, communication, 
and design, students’ perceptions of their ability in design have begun to increase over time, while 
those in communication and teamwork did not change, as shown in Figure 5(b). Note that the rela¬ 
tively flat self-perceived abilities at graduation in communication and teamwork do not agree with 
the alumni survey data discussed earlier. It may be that students have difficulty self-judging their 
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Figure 5. Changes in students’ self-reported ability normalized to the control responses 
on coursework are shown in (a), while the design category is subdivided into design, 
communication, and teamwork in (b). The arrows represent the start of the planning 
(dashed) and implementation (solid) phases of the project. 
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abilities in teamwork since their basis for comparison may be peers who also have increased experi¬ 
ence. Similarly students may judge teamwork skills on a small number of bad experiences that arise 
from their own lack of ability. 

To measure conceptual understanding at graduation, a summative examination that draws ques¬ 
tions from multiple, validated concept inventories is given at the end of the first, required capstone 
design course, taken in the senior year. Recently concept inventories have been developed for various 
topics in electrical engineering. ES21C combined eight inventories into a single examination to test 
students’ understanding of fundamental concepts underlying electrical engineering in each of the 
different curricular areas of specialization taught by in the degree program. Approximately eight 
questions from each of seven separate inventories have been chosen as representing electrical en¬ 
gineering fundamentals. Faculty across the program ranked questions on a 1-5 scale on two criteria: 
1) questions every student should be able to answer correct if they have understood basic concepts, 
and 2) questions that cover concepts that all students are taught at least once. The questions that 
were deemed easiest and most heavily covered were chosen for the exam. The test consisted of 
52 questions given as a two-hour, closed-book final examination. All students finished within the 
two hour time limit. Sample sizes ranged from N = 51 to N = 12 students for the data presented. Of 
the eight areas covered by concept inventories, questions were separated by those that addressed 
topics covered in courses that had undergone reform under ES21C, and those that were not directly 
involved in reform. Since it wasn’t possible to perform on a one-to one map between questions and 
specific courses, there is likely some overlap between the groups. Note also that unlike the usual 
format of concept inventories, the test was not given in a controlled pre-post format due to the 
wide range of questions. 

Comparisons between aggregate scores (average of about 25 questions each) for ES21C- 
influenced and non-ES21C courses are shown in Figure 6 over a four year period beginning at the start 
of the project. Data is not available for earlier years. The data indicates that compared to non-ES21C 
courses, those courses which adopted the ES21C model show an increase in student’s conceptual 
understanding over the four year period of the project. It should be noted that the composite con¬ 
cept inventory was given in the senior year and most courses which covered these topics are taken 
in the junior year. Thus it is not clear if the results are due to increases in conceptual understanding 
or a better retention of concepts. As with previous results, the breadth of the reform effort makes 
it impossible to say that the observed trend was caused by ES21C, but improvement in conceptual 
understanding are correlated with the ES21C teaching method. 

To determine if ES21C’s increased influence on engineering procedural knowledge impacted 
more traditional measures of student ability, the results of the Fundamental of Engineering (FE) 
exam were tracked over the project. The FE exam asks 120 questions covering the basics of general 
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Figure 6. Average score over time on composite concept inventory given in the senior 
year for topics in ES21C and non-ES21C courses. 

engineering and subject specific topics in two four-hour sessions. Students self-select whether 
to take the FE exam, with the program helping offset the cost of the examination. Generally be¬ 
tween four and ten students take the exam each semester. To ensure that these students form a 
representative sample, the GPA of all students who take the exam is compared to all students in 
the program. The FE exam scores on all exam categories have been tracked over the duration of 
the project using the scaled-score method (LeFevre, et al., 2005). Scaled scores of ±1 correspond 
to variations of one standard deviation from the national mean. Despite the increased focus on 
design and projects there is no discernable loss on more traditional learning since scores on the FE 
exam have stayed relatively constant within errors of the measurement. Looking within categories 
of the subject exam, three categories can be mapped to ES21C courses, while six categories reflect 
content that is not directly taught in ES21C courses. Two of the three categories averaged for the 
ES21C data points were reformed in the initial phase of the project in 2003. Comparing ES21C to 
non-ES21C courses, there was a slight downward trend in ES21C courses, compared to other courses 
on the subject exam. A linear fit shows a 20% decrease over five years, as opposed to a 5% decrease 
for non-ES21C courses, but the drop is not statistically significant and the overall scores for both 
groups are comparable within the error. There is additionally more variation in year-to-year scores 
for ES21C courses, but it is not clear whether this is due to changes in learning outcomes or simply 
since this is a composite of three rather than six categories. It should be noted that due to the vari¬ 
able, and relatively small number of students taking the exam every semester that FE exam scores 
tend to fluctuate semester-by-semester. 
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To measure changes in student metacognition, the Metacognitive Awareness Index (MAI) was 
given in the sophomore and senior years (Schraw and Dennison 1994). The MAI has 52 questions 
that measure metacognition on two scales—knowledge of cognition (25 questions) and regulation of 
cognition (27 questions)—which are intercorrelated at r = 0.45. Responses are on a five point Likert 
scale with 1 corresponding to not at all true of the respondent and five very try of the respondent. 
Both sophomores and seniors mean scores were 3.8 on the knowledge of cognition and 3.3 on the 
regulation of cognition. There were no changes in MAI scores from the sophomore to senior years 
or longitudinal changes over time on either scale of the MAI. 

In summary, data that measures overall outcomes, rather than specific course or learning ob¬ 
jectives, paints a picture, while still incomplete, of students who encounter design problems more 
often and have multiple experiences with teamwork and communication in design, and value these 
experiences following graduation. This additional experience has been purchased at the expense 
of broad, preparatory courses, however long-term measures do not indicate any significant de¬ 
crease in students' ability to solve traditional problems as measured by FE exam scores. Preliminary 
data shows that ES21C is also having a positive impact on students' conceptual understanding. It 
should be noted that the observed trends do not necessarily prove causation. A degree program 
is constantly changing and evolving, and other changes in how students learn cannot be ruled out 
as causing the observed changes to student learning. 


DETOURS ALONG THE ROAD TO REFORM 


In an ideal world a program for course reform that was based on research in how students learn 
and liberally lubricated with NSF funding would be embraced by faculty, implemented in courses, 
and student learning would improve. In reality there were unforeseen roadblocks on the path to 
reform. This section briefly discusses some of these obstacles and the changes to the project direc¬ 
tion needed to overcome or bypass them. 

Many of the unanticipated obstacles to reform that impeded ES21C in the first two years came 
from groups or individuals and could have been avoided with some forewarning. A lack of interest 
in program reform by some faculty was anticipated; the course-focused rather than curriculum- 
focused structure of ES21C was designed to bypass faculty who chose not to participate. The project 
did not, however, anticipate the active opposition to reform by a small number of individuals. The 
reasons for opposition to reform were difficult to clarify, but primarily arose from personal beliefs 
about teaching based on individual experiences, either good or bad. Other opposition arose from the 
perception that the six step model was prescriptive and that faculty were being forced to change 
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their teaching, often in ways that ran counter to deeply held beliefs. The project also underestimated 
communication barriers between faculty who were conversant with engineering education research 
and those who were not. Some ES21C participants in OSU’s College of Education are actively inves¬ 
tigating the role of faculty beliefs in helping or hindering reform (James, et al. 2007). 

The prescriptive view of reform likely arose from the conception, often implicit in engineering 
education literature, that there are “better” or “worse” ways of teaching. Many faculty who were 
engaged in ES21C were looking for a better way of teaching while those who opposed reform 
thought their teaching was being characterized as “worse”. One of the changes in direction made 
during the project was to relax initial requirements of adherence to the strategic model outlined 
previously section. It was found that identifying learning goals for classes first, then suggesting 
changes to course structure or pedagogy from the elements of the ES21C strategy that supported 
faculty-identified goals was a more effective approach than starting with changes to pedagogy. 

At the start of ES21C large meetings of all participants were held and education experts were 
invited to discuss effective teaching techniques. These meetings were generally unsuccessful; most 
faculty did not learn well from lecture. As the project evolved, the strategy shifted to holding one- 
on-one or small group meetings between faculty and a PI immersed in engineering education and 
the philosophy of ES21C. This approach, while demanding of Pi’s time, was more effective at creating 
effective and sustainable change. 

The project also failed to anticipate the difficulties that arose from fundamental differences in 
structure between the traditional, knowledge-based and reformed, developmental courses. Tradi¬ 
tional courses are highly linear since information and concepts build sequentially. Textbooks serve 
as signposts for different pathways to knowledge acquisition. In the ES21C model, course activities 
follow more of a design paradigm without a clearly defined, linear path. In ES21C what is learned 
depends upon the project used to scaffold learning, content is classified by a cognitive process level, 
and sequencing content is dependent on the chosen pathway to problem solution. Since this is the 
approach to learning that is often followed in research, it was anticipated that it would be familiar 
to faculty. Participants had difficulty, however, transferring their own metacognitive processes to 
create learning opportunities for students. 


CONCLUSIONS AND SUSTAINABILITY 


With, at the time the article was submitted, just over one project year remaining, Engineering 
Students for the 21st Century continues to change the culture of OSU’s ECE department. While the 
pace of change has been slower than expected, the broad vision set forth by the project is slowly 
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being implemented. As intimated by the Machiavelli quote at the start of this article, such change 
has not been easy and it is an open question whether the pace of change, or even the cultural 
change itself will be sustainable in the years following conclusion of the project. This section first 
comments on sustainability then concludes with several observations that may be valuable to other 
programs that seek to initiate wide-spread and aggressive reform efforts. 

Engineering Students for the 21st Century, as initially conceived, focused on changing ten courses 
to engender program reform. In retrospect, the decision to implement change in individual courses 
likely arose from the predominance of course-level studies in the STEM education literature that pro¬ 
vided clear guidance for reform at this level. While ES21C’s emphasis on courses is robust, it will face 
pressures caused by faculty turnover, changes in course assignments, and shifts in faculty interest 
over time. To be sustainable, ES21C is first changing courses then integrating these changes into the 
degree program. Integration into the degree program has also been pursued by developing depart¬ 
ment-wide infrastructure and resources to support design as outlined earlier. Changes to department 
infrastructure support sustainable reform since such infrastructure is often maintained by staff with 
clearly defined responsibilities. ES21C is also examining ways to implement faculty reward structures, 
but since this requires cultural change at an administrative level, progress has been slow. 

Another way that ES21C is trying to make reform sustainable is through synergistic efforts with 
external program evaluation such as ABET Some faculty think of ABET, if they think of ABET at all, 
as eleven vaguely worded outcomes, the (a)-(k) (ABET 2008). The inherent vagueness of (a)-(k), 
however, allows programs to interpret and personalize outcomes, as well weigh outcomes to sup¬ 
port program reform. ES21C has aligned reform goals with ABET accreditation goals, engaged key 
faculty in the accreditation process, and developed common sets of evaluation metrics. Another 
pathway to sustainability integral to ES21C are the significant efforts to engage graduate students 
and young faculty in reform. Many of these efforts have occurred through partnerships with other 
organizations within the university rather than starting new efforts within ECE. 

In retrospect, what advice can Engineering Students for the 21st Century offer to other programs 
interested in reform of engineering education, particularly programs without the generous funding 
provided by the National Science Foundation? First, recognize that the time constant of reform ef¬ 
forts is much longer than many other engineering projects. Most faculty participating in ES21C are 
also engaged in technical research projects that last from several months to three years. Program 
reform, which involves cultural change, has a time constant closer to a decade. Like physical systems, 
trying to drive a degree program at a rate much faster than the natural response time is extremely 
difficult and inefficient. A key first step is to try reform on a small scale and be willing to accept poor 
outcomes while experimenting. Several years of missteps are followed by successes by the pioneers; 
recognizable by the arrows in their backs. Initial successes are eventually discussed, mimicked with 
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varying degrees of success, then adapted to meet faculty’s beliefs and needs. To misquote Paul 
Samuelson, over time and funeral by funeral reform advances. It is critical, in our experience, that 
reform efforts have a clear, simple, and easy to communicate vision to serve as a guide star in order 
to maintain direction over the long term. ES21C did not initially realize this fact, and consequently 
has struggled to communicate the project vision to faculty. 

One important observation is that the effectiveness of ES21C increased dramatically when 
infrequent large, formal meetings were replaced with more informal one-on-one or small group 
meetings held on a regular basis. While this placed significant time burdens on some faculty, it 
resulted in more lasting and significant change. Thus a second recommendation for others seek¬ 
ing departmental reform is to hire or support faculty who have backgrounds both in engineering 
education and technical fields. These faculty, who have credentials in technical research, serve as 
ambassadors to other faculty. It is important to note that such positions have been found to incur 
significant personal risk (Bush et al. 2008). 

At the start of Engineering Students for the 21st Century the participating faculty were aware 
that students needed more individual choice in some aspects of their education in order to develop 
into engineers. In retrospect it is clear that ES21C failed to identify the same need in faculty. Some 
of the rocky start of the project could have been avoided had the primacy of faculty beliefs over 
engineering education research results been recognized. The importance of faculty beliefs is likely 
the root cause that working individually with faculty proved so effective. 

How can programs interested in reform and faculty who have nagging doubts about how effec¬ 
tively we are educating the next generation of engineers adopt the ES21C model to their institutions 
without the generous support provided by NSF? While it is tempting to end this retrospective by 
outlining a “magic bullet” formula or procedure for reform, there are no magic bullets. The pre¬ 
scriptive approach was unsuccessful in the planning phase of ES21C, replaced with a more strategic 
approach in the implementation phase, then the strategy itself relaxed as the project evolved. It is 
more correct to think of program reform as a journey that is aided by guideposts, but is ultimately 
up to the traveler to undertake. If the program were, however, to redo the reform project with fewer 
resources, there are several lessons that would help prepare for the journey: 

• Revamp existing labs and design courses by making changes to the learning environment 
that give students more freedom and flexibility. An investment of several hundred dollars of 
e-commerce software enabled ES21C to develop on-line catalogs that have made it easier for 
students to self-schedule time to work on problems. 

• Create classes in which knowledge acquisition occurs outside the classroom and application 
of what is learned is done in class under guidance of the instructor. Lectures can be moved 
to video formats- a tablet computer, video capture software, and YouTube allow any faculty 
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member to self-publish on-line content for under $2000. Once on YouTube students have 
access to material at need, as do others across the globe. 

• Give students opportunities to talk through complex engineering problems on their own terms 
without being told what they need to learn. Case studies, when properly implemented, are a 
cost- and time-effective way to give students a voice. 

• Consider how the degree program could be structured so that course content is not the pri¬ 
mary organizing principle. Content is important, it defines the degree, but content needs to 
be combined with application. The skills, attitudes, and experience students need to develop 
need to be identified separately from content. The engineering taxonomy ( hyperlink ) intro¬ 
duced earlier is one tool that can be used in reflecting on programs. 

• In moving from a content-driven to development-driven program, ES21C has found it especially 
valuable to contextualize work around the design cycle shown in Figure 3, particularly for 
courses early in the program. All student work is related to the design cycle to help students 
understand how what they are doing fits into a larger picture. 

• Know thyself. This aphorism should be interpreted two ways. First, clearly define the overall 
goals and direction for your program and how students should change as they advance to¬ 
wards a degree. While many programs dread six-year accreditation, ABET can be a powerful 
force for change when it is used as a goals-driven process rather than an externally imposed 
mandate. Second, recognize that faculty members’ beliefs about teaching and learning likely 
arose from their personal experiences and preconceptions and do not universally apply to all 
students. Be willing to attempt teaching experiments even if they go against some beliefs in 
order to expand the repertoire of pedagogies available in the program. 

• Develop in-house expertise in engineering education. While most faculty teach, few have much 
depth in the science of learning. Having a faculty member in the department who develops 
expertise in engineering education not only brings new perspectives, but can help externally 
fund reform efforts by submitting competitive proposals. 

To conclude on a resounding note, it is tempting to end this paper with the claim that our program 
is better as a result of reform. As discussed previously, however, the concepts of “better” and “worse” 
lead to dangerous territory. Rather it can be concluded that our program is different, and the dif¬ 
ferences support the new conception of engineering education put forth by a multitude of national 
panels (National Research Council 2004, 2005, 2007, 2009 Hyperlink to these resources ). Creating 
these differences has come at a significant cost of both resources and time. The authors, as do other 
participants, suffer moments of self-doubt in which we wonder whether the NSF resources put at 
our disposal have been used effectively and if the effort at reform has been worthwhile. The reflec¬ 
tion and analysis required to write this paper have, however, reaffirmed the belief that department 


32 


SUMMER 2011 







ADVANCES IN ENGINEERING EDUCATION 

Engineering Students for the 21 st Century: Student 
Development Through the Curriculum 



reform is not only worthwhile, but vital to our well-being as a discipline and as individuals. Reform 
requires us to listen to what author Parker Palmer calls our “inner voice”. The journey is difficult; as 
Machiavelli intimated, there is much resistance to change, compounded by the difficulty of “proving” 
the value of reform. While there will always be doubt whether the journey of reform is worthwhile, 
as with Tinkerbell in the stage version of Peter Pan , doubt can itself be fatal. By looking deep within 
ourselves, by reflecting on our own values and those of our community, we can create ways for our 
students to discover for themselves the value of engineering, ways of developing those who will 
carry out the great works of this age. Context and culture are important; without stories, without 
tales of heroes, without a guiding mythology it is hard for many students to discover passion in a 
dispassionate discipline. At the end of the degree program is it more important for our students to 
share our knowledge or our culture? 
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